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ABSTRACT 

Background No standardised set of quality 
measures associated with transitioning complex- 
care patients across the various healthcare 
settings and home exists. In this context, a 
structured panel process was used to define 
quality measures for care transitions involving 
complex-care patients across healthcare settings. 
Methods A modified Delphi consensus 
technique based on the RAND method was used 
to develop measures of quality care transitions 
across the continuum of care. Specific stages 
included a literature review, individual rating of 
each measure by each of the panelists (n=1 1), 
a face-to-face consensus meeting, and final 
ranking by the panelists. 

Results The literature review produced an initial 
set of 1 19 measures. To advance to rounds 
1 and 2, an aggregate rating of >75% of the 
measure was required. This analysis yielded 
30/1 19 measures in round 1 and 1 1/30 measures 
in round 2. The final round of scoring yielded the 
following top five measures: (1) readmission rates 
within 30 days, (2) primary care visit within 
7 days postdischarge for high-risk patients, 
(3) medication reconciliation completed at 
admission and prior to discharge, (4) readmission 
rates within 72 h and (5) time from discharge to 
homecare nursing visit for high-risk patients. 
Conclusions The five measures identified 
through this research may be useful as indicators 
of overall care quality related to care transitions 
involving complex-care patients across different 
healthcare settings. Further research efforts are 
called for to explore the applicability and 
feasibility of using the quality measures to drive 
quality improvement across the healthcare system. 

INTRODUCTION 

Patients with complex needs associated 
with their multiple diseases, comorbidities 



and conditions, frequently require care in 
multiple settings, and are particularly vul- 
nerable to poorly executed transitions in 
care. 1 2 Complex-care patients are man- 
aging multiple chronic diseases, taking 
several medications, may have limited 
social supports or suffer from dementia, 
mental illness or addiction. This complex- 
care patient cohort is predicted to grow 
substantially as the population ages and 
chronic diseases continue to be a major 
expenditure of healthcare costs. 3 Poor 
transitions often result in unnecessary 
adverse patient outcomes and additional 
healthcare spending. 4 

Care transitions refer to a set of actions 
designed to ensure the coordination and 
continuation of appropriate healthcare 
provision as a patient transfers between 
different levels of care in the same clin- 
ical setting or different locations. 5 
Current literature examining transitions 
in care has mainly focused on transfers 
within acute-care organisations and to 
the home setting. 6-8 However, complex 
patients, at any age, may experience tran- 
sitions across the continuum of health 
services from acute care to multiple 
healthcare settings, such as complex con- 
tinuing care rehabilitation centres, long- 
term care, home, or possibly from acute 
care teaching hospitals to community- 
based hospitals. 

The importance of gaining insight into 
transfers of complex-care patients across 
the lifespan and between settings is a crit- 
ical and timely policy issue. At present, 
there is no standardised set of quality 
measures associated with transitioning 
complex-care patients across the various 
healthcare settings and discharged and/or 
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readmitted to/from home. 9 This is an important gap 
to address, as quality indicators are increasingly being 
employed to assess and improve quality of care in 
many healthcare settings. 10 Therefore, the purpose of 
this study was to define quality measures for care tran- 
sitions involving complex-care patients across the con- 
tinuum of care through a structured panel process. 
Our focus was to develop indicators that would be 
useful for accountability purposes as well as ones that 
had the potential for broader quality improvement 
(QI) efforts. 



METHODS 

Overview of modified Delphi panel process 

A modified Delphi consensus technique based on the 
RAND method was used to develop measures of 
quality care transitions across the continuum of 
care. 11 The panel process was built on a technique 
that was shown to be effective for developing quality 
indicator measures in healthcare. 12-18 For this study, 
multiple stages of the modified Delphi technique 
(literature review, individual rating, face-to-face con- 
sensus meeting, and final ranking) were used (see 
figure 1). Key to this methodology is engaging in mul- 
tiple rounds of feedback from the expert panel using 
both anonymous surveys and round table meeting 
techniques to allow for optimal, unbiased expression 
of opinions. 
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Initial work: literature review and selection of initial set of 
measures 

The first step in the Delphi technique is to engage in a 
structured review of the literature. The intent of this 
review was to produce a list of measures that had 
been reported in the literature that would then be pre- 
sented to a panel of experts to review and rate. This 
literature search was undertaken in September 2011 
using OVID Medline with the following key words 
with the number of abstracts reported: Quality 
Assurance, Health Care or Quality of Health Care; 
Health Transition (4 217 740); Patient Transfer, 
Patient Discharge, Continuity of Patient Care, Health 
Services for the Aged, Delivery of Health Care, 
(714 391) Home Care Services, Chronic Disease, 
Aged, Long Term Care (2 280 704); Primary Health 
Care, Delivery of Health Care, Patient Transfer, 
Continuity of Patient Care (723 017); Health 
Transition (650); Outcome and Process Assessment 
(Health Care) and Outcome Assessment (Health 
Care) (571 586). These search terms were used as 
they were identified by the research team and key sta- 
keholders as key terms for care transitions. 

Other databases beyond OVID Medline were not 
included as the initial search yielded a significant 
number of abstracts. The search was limited to those 
written in the English language (1675) and from 2000 
to 2011, as any previous studies would have been 
included in literature syntheses during this time frame. 
Other limits included articles that were clinical 
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Figure 1 Illustration of modified-Delphi process. 
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conferences, lectures and legal cases, newspaper arti- 
cles, commentaries and bibliographies/biographies 
(1658). The final limit was to include meta-analysis 
and reviews only (76). These 76 abstracts were 
reviewed by three members of the research team to 
determine which ones reported on outcome measures 
associated with care transitions. This review yielded 23 
articles including two meta-analyses 19 20 and 21 litera- 
ture reviews, 21-41 This review was augmented with 
other literature known to the research team, or were 
references included in the reference lists of the litera- 
ture review of primary studies. 5 41-51 

Additionally, a search of targeted journals and grey 
literature was conducted in early January 2012 that 
yielded 14 other documents that included evidence 
for transitions of care outcome measures. 9 52-64 This 
second search was undertaken for several reasons. 
First, some of the documents reviewed referred to 
other references that were not picked up in the initial 
literature search. 9 51-58 61-64 Second, shortly after the 
literature search was conducted, a report was released 
on avoidable hospitalisations. 52 Third, the research 
team became aware of additional interventions studies 
published in early 20 12. 59 60 Of the literature that 
was identified, all the studies were included in the 
next phase of the project measure development. Our 
research team developed an initial survey of potential 
measures from all measures of quality of transitions of 
care for complex-care patients listed in the final set of 
documents reviewed. The list of measures (n=119) 
were categorised under the following headings: health 
service usage; patient outcome; patient satisfaction 
and experience; provider satisfaction. Once the initial 
set of measures was determined, the research team 
identified potential panel experts to be engaged in 
this project. 

First-round ratings 

Panel member selection. The research team developed 
criteria for panel member selection in partnership 
with key stakeholders (Canadian Patient Safety 
Institute, Accreditation Canada and Patients for 
Patient Safety). Criteria for selecting panel members 
included having an interest in care transitions, patient 
safety, QI, patient care experience and related research 
initiatives. Additionally, key stakeholders (listed above) 
were asked to provide names of experts and organisa- 
tions to include as potential panel members. From 
this, an initial list of panel members was created by 
the research team which had expertise in care transi- 
tions and measure development. There was originally 
a panel of 16 who committed to being part of the 
research project, with the final panel of 11 partici- 
pants representing a broad scope of expertise in care 
transitions. The withdrawal of the five participants 
was due to scheduling conflicts. The final panel con- 
sisted of clinicians (medicine, pharmacy and nursing) 
including a general internist with patient safety 



expertise, researchers with measure development 
expertise and one who had extensive research back- 
ground in care transitions. Both a national and provin- 
cial level policy maker were part of the panel as were 
decision makers from a variety of sectors, including 
acute care, home care, long-term care, complex con- 
tinuing care/rehabilitation care and primary care. 
Stakeholder organisations and patient advocacy group 
were also represented on the final panel. 

First round review survey. In the first round of the 
Delphi approach, the panel was sent the questionnaire 
electronically 4 weeks prior to the face-to-face panel 
meeting and asked for input to identify measures 
worthy of consideration. A reminder email was sent 
to panelists 2 days before, and on the day of the dead- 
line. We chose to include all relevant measures 
(n=119) in the survey (see online supplementary file 
for list). The survey package included an overview of 
the literature review process, the list of measures with 
definitions for calculations as described in the litera- 
ture (where provided), and literature summaries of 
each of the articles and reports. Additionally, an over- 
view of the rating scale was provided and outlined in 
table 1. 

Specifically, panel members were asked in the 
survey to evaluate the proposed measures using the 
Quality Measures Attributes developed by the Agency 
for Healthcare Research and Quality (AHRQ). 65 This 
framework groups measures according to four broad 
conceptual areas: (1) importance of the measure; 

(2) scientific soundness of the measure — clinical logic; 

(3) scientific soundness of the measure — measure 
properties; (4) feasibility of the measure. Given the 
focus of our study was to identify quality measures 
that may in the future be useful for accountability pur- 
poses and to make system improvements for complex- 
care patient transitions, we also included two add- 
itional criteria for evaluation including the usefulness 
of the measure: (1) to drive QI efforts and (2) for 
accountability purposes in public reporting (PR). We 
included the means for the two latter measures, in 
addition to the overall mean average of the six evalu- 
ation criteria, due to our interest in developing mea- 
sures for accountability purposes and QI efforts. The 
panel used a nine-point Likert scale which considered 
scores of 1-3 as strong disagreement, 7-9 as strong 
agreement and 4-6 as uncertain agreement. An overall 
total score that averaged all six evaluation criteria was 
calculated alongside the score for the usefulness of the 
measure to drive QI efforts and the usefulness of the 
measure for accountability purposes in PR. Panel 
members were also provided with the opportunity to 
suggest additional measures prior to the in-person 
meeting. To ensure measures developed through the 
panel are useful and relevant over time, panelists were 
encouraged to consider the extent to which identified 
measures can be linked to currently employed quality 
of care processes as well as what they view as future 
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Table 1 Rating matrix domains Desirable Attributes of a Quality Measure — Definitions: The research team has selected the Quality 
Measures Attributes developed by the Agency for Healthcare Research and Quality (AHRQ) to guide the rating matrix of the Transitions of 
Care Outcomes/Measures. Table 1 provides a description of each of the desirable attributes of a Quality Measure. For further information 
of AHRQ's Desirable Attributes of a Quality Measure visit: http://www.qualitymeasures.ahrq.gov. 

AHRQ Desirable Attributes of a Quality Measure — Definitions 



Quality measure domain 



Criteria description 



Importance of the measure Relevance to stakeholders — the topic area of the measure is of significant interest, and financially and strategically 

important to stakeholders (eg, patients, clinicians, purchasers, public health officers, policy makers) 
Health importance — the aspect of health that the measure addresses is important as defined by high prevalence or 
incidence, and/or a significant effect on the burden of illness (ie, effect on the mortality and morbidity of a population) 
Applicability to measuring the equitable distribution of healthcare (for healthcare delivery measures) or of health (for 
population health measures) — the measure can be stratified or analysed by subgroup to examine whether disparities in 
care or of health exist among a diverse population of patients 

Potential for improvement — there is evidence indicating a need for the measure because there is overall poor quality or 
variations in quality among organisations (for healthcare delivery measures) or overall poor quality of health or 
variations in quality of health among populations (for population health measures) 

Susceptibility to being influenced by the healthcare system — for healthcare delivery measures, the results of the 
measure relate to actions or interventions that are under the control of those providers whose performance is being 
measured, so that it is possible for them to improve that performance. For public health measures, the results should 
be susceptible to influence by the public health system 

Explicitness of evidence — the evidence supporting the measure is explicitly stated 

Strength of evidence — the topic area of the measure is strongly supported by the evidence, that is, indicated to be of 
great importance for improving quality of care (for healthcare delivery measures) or improving health (for population 
health measures) 

Reliability — the results of the measure are reproducible for a fixed set of conditions irrespective of who makes the 
measurement or when it is made; reliability testing is documented 

Validity — the measure truly measures what it purports to measure; validity testing is documented 
Allowance for patient/consumer factors as required — the measure allows for stratification or case-mix adjustment if 
appropriate 

Comprehensible — the results of the measure are understandable for the user who will be acting on the data 

Explicit specification of numerator and denominator — a measure should have explicit and detailed specifications for the 
numerator and denominator; statements of the requirements for data collection are understandable and implementable 
Data availability — the data source needed to implement the measure is available and accessible within the timeframe 
for measurement. The costs of abstracting and collecting data are justified by the potential for improvement in care or 
health 

To ensure measures developed through the panel are useful and relevant over time, you as a panellist are encouraged to consider the extent to which 
identified measures can be linked to quality of care processes currently, as well as your view as future areas that necessitate attention related to improving 
transitions of care for medically complex patients across the continuum. 

In addition to the four AHRQ attributes of a quality measure, two additional criteria for evaluation will be used: 

► Measuring this is useful in driving transitions of care quality improvement. 

► Measuring this is useful for accountability purposes such as public reporting. 

Instructions — You will be asked to rate each of the transitions of care outcomes/measures and/or associated contextual variable identified from the 
literature review using the following rating matrix (domains of quality measures) on a 9 point Likert Scale ranging from 1 strongly disagree to 9 strongly 
agree. For example, choosing strongly agree on the "Importance of Measure" reflects your agreement about this particular measure in relation to the 
criteria description in the AHRQ Desirable Attributes of a Quality Measure provided in Table 1. 



Scientific soundness: clinical 
logic 



Scientific soundness: measure 
properties 



Feasibility 



Rating matrix domains 



Strongly disagree 



Strongly agree 



Measure 
Importance of measure 
Scientific soundness: clinical logic 
Scientific soundness: measure properties 
Feasibility 

Useful to drive quality improvement 
Useful for public reporting 



1 



AHRQ, Agency for Healthcare Research and Quality. 



areas that necessitate attention related to improving 
transitions of care for complex-care patients across 
the continuum. Consensus on these surveys was 
determined to be reached upon a 75% agreement 



\16 17 



(standard percentage in other studies) x ~ among 
panel members on the three scores (mean total score 
for all six rating criteria; mean score for usefulness to 
drive QI; and mean score for usefulness for PR). 
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Second-round panel meeting 

For the second round, panel members met in person 
for a 1-day facilitated structured discussion of the 
results compiled from the first round of the question- 
naire. At the conclusion of the discussion, members 
were asked to complete a second round of the ques- 
tionnaire using the same evaluation criteria and rating 
scale. After completion of all discussion and question- 
naires, panellists were also asked to anonymously 
comment upon their experience regarding the overall 
consensus process as a mechanism to ensure the 
process was fair and unbiased. 

The final results were tabulated using the same con- 
sensus agreement score (>75%). The panel was then 
contacted once more to provide a priority ranking 
(from 1-11) among the final selected measures. 
Again, individual responses remained anonymous to 
all other panel members. The mean frequency of 
the rankings of the set of measures was calculated to 
yield the final set of measures. Additionally, panellists 
were asked for their personal views of where the pri- 
ority for measure usage and development should be 
regardless of current data availability and ease in 
feasibility. 

RESULTS 

First round ratings 

From the initial set of 119 measures, 30 measures 
received an aggregate rating of >75% (7/9 or more) 
across one or more of the total score of the six 
domains, useful to drive QI, and useful for the PR 
domain (see table 2). 

Second-round ratings — panel 

Analysis from the second round of panel member 
ratings revealed a set of 11 measures (see table 3). 
This included seven measures that received an aggre- 
gate rating of >75% (7/9 or more) in all three scores 
— total score of the six domains, useful to drive QI 
and useful for the PR domain; three measures 
received an aggregate rating of >75% (7/9 or more) 
total score of the six domains, and useful to drive QI 
score and between 6 and 6.99 on the useful for PR 
domain; and one measure received an aggregate rating 
of >75% (7/9 or more) in the useful to drive QI. 
Totally, 11 out of the original measures were selected 
to go to the final round of rating, with wording 
changes to four of the measures (medication reconcili- 
ation, time from discharge to homecare nursing visit 
for high-risk patients, mortality rate and discharge 
summary), and readmission rate stratified by three 
time frames — within 72 h, 30 days and 90 days. 

Totally, 21 out of the original 30 measures were 
eliminated in this round. Analysis of the key discus- 
sion points at the panel meeting and hand written 
notes from the participants revealed that although 
these measures were important quality measures of 
care, they were not perceived collectively to be a 



quality measure of a care transition. For example, 
several patient clinical outcome measures (eg, nosoco- 
mial infections, pressure ulcers, pain, functional status 
and medical errors) were viewed as not being an 
outcome of an ineffective care transition. Other key 
issues identified with those measures that were elimi- 
nated included the perception that the measures were 
unclear or too broad to be an effective indicator of 
the transition (eg, potentially avoidable hospital 
admissions and hospital discharge process). Other key 
discussion points included a focus on patients and 
family perspectives for what constitutes quality care 
transitions for complex-care patients across the care 
continuum. The panel members recommended 
further work be undertaken to develop more system- 
level measures that would capture complex-care 
patient and family members' satisfaction and experi- 
ences along the care continuum. 

A postmeeting survey found unanimous agreement 
from the participants that their opinions were valued, 
the process was fair, and that no one altered their 
responses or opinions as a result of feeling intimidated. 

Final review ratings 

When asked to rank the most important final indica- 
tors with respect to overall score, QI and PR purposes, 
with the exception of readmission rates within 30 days 
as the number one priority and multiple psychiatric 
readmissions in 30 days as the last priority, there were 
mixed responses regarding where the priority focus 
should be (see table 4). Ranking the measures using the 
overall total score yielded the following top five mea- 
sures are readmission rates within 30 days, primary 
care visit within 7 days postdischarge for high-risk 
patients, medication reconciliation completed at admis- 
sion and prior to discharge, readmission rates within 
72 h, and time from discharge to homecare nursing 
visit for high-risk patients. 

DISCUSSION 

Through a structured, modified Delphi panel process, 
a series of quality measures for care transitions involv- 
ing complex-care patients across the continuum of 
care were delineated. Our inclusion of health services 
researchers, decision-making organisations, frontline 
clinicians, (physicians, nurses and pharmacists) and a 
patient representative increased the likelihood that the 
selected quality measures are scientifically sound and 
have broad applicability and feasibility. 10 18 Our final 
list of quality indicators included outcomes from the 
three domains of focus including nine health services 
measures (readmission rates within 30 days, primary 
care visit within 7 days postdischarge for high-risk 
patients, medication reconciliation completed at 
admission and prior to discharge, readmission rates 
within 72 h, time from discharge to homecare nursing 
visit for high-risk patients, discharge summary all 
medications and follow-up appointments given to 
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Table 2 Results from prepanel questionnaire of measures achieving at least 75% consensus 



Measure 



Mean 

total 

score 



Mean 
drive 
Ql 



Mean 
public 
reporting 



Health services usage measures 








Readmission/readmission rates (72 h, 7, 30, 90 days) 


7.76 


7.73 


7.6 


FD \/ki"K/Fn ^Hmkdnn r^tp*; 


7.43 


7.26 


7.29 


Mine Tiom aiscnarye to nornecare nursing visit toi nign-nsK panenis 


7 c;7 


7 Q3 
/ 


7 3£ 


Primary care visit within 7 days postdischarge for high-risk patients 


7.29 


7.64 


7.07 


Medication reconciliation completed prior to discharge 


7.69 


8.08 


7 


r-N 1 1 /II 1" j." 1 £ II " i i \ " i i " i 

Discharge summary (all medications and follow-up appointments) given to patient 


7.7 


8.14 


7.31 


Discharge summary (all medications and follow-up appointments) given to primary healthcare provider 


7.91 


8.29 


7.71 


Ambulatory sensitive conditions hospitalisation/ambulatory care visits 


7 


7.63 


6.94 


Healthcare connected for unattached patients 


7.08 


7 


6.85 


Hospital discharge process (discharge planning, plan of care, postdischarge support, hospital assessing 


7.07 


7.5 


6.71 


needs and using discharge plans) 








n> ■ 1 i * i * i ■ . . . i 

Discharge medication list sent to pharmacy 


7.04 


7.07 


6.29 


Continuity of the medical records upon hospital discharge to the outpatient setting 


7.27 


7.46 


6.77 


Clinical performance (screening, immunisation, chronic care after acute myocardial infarction, diabetes 


7.15 


7.15 


6.62 


mpllitiK rnntrnl h\/nprtpn«;inn rnntrnh 

IIICIIIIUj V.UIIUUI, 1 IV UCI LCI 1 JlUI 1 v_UIIUUI/ 








Potpntiallv avnidahlp hosnital adrnissinns 

1 \J LCI 1 LI a 1 1 y CI VV_/IVJCIL/IC 1 IUjUI LCII C4\J 1 1 1 1 jjI ui \d 


6.76 


6.67 


7.27 


Miiltinlo no/rhi^trir rp^Hmiccinnc 
IVIUILipic UbyU llaLI IL 1 caUl 1 lloolUI 1 j 


Q1 


U. JJ 


7 11 

I.LI 


F^rk/ rnll^hnr^tix/p r^rp/mi iltiHicrinlinarv P3rl\/ cimnnrtpH Hicrh^rnp 
Ldlly LUIIdUUI all Vc Ldlc/1 1 IUI UUIbUUIII Idl y cdlly jUUUUI LcU UloLMdlLjC 


n Q 

U.J? 


U.JJ 


1 31 


Pstipnt pdurstinn/splf-rn^nsnprnpnt sunnort 

1 QLICIIL CVJUCCILIV-/II/JCII IIICIIICIVJCIIICIIL JUUUUI L 


6.93 


6.21 


7.07 


P^tipnt Ip\/pI mppiciirpc rlinir^l 

rdLlclll level MlcdjUlcj UIIIILal 








Mncnrnmipil infprtinn 

IMUoULUI 1 lldl MIlCLUUM 


7.19 


7.25 


7.17 


Mnrtalitw/Hp^th/r;}! kp nf Hp^th/rkk' nf Hp^th 
IVIUI LdllLy/ucdLl 1/LdUjC UI UCdLII/lljlN. ui Ucdlll 


7.24 


7 


6.85 


Functional status/nhvsiral funrtinn/irnnrnvpinpnt 

1 UIICLIVJIIQI _) La LU 31 U 1 1 y 0 1 v_a 1 1 U 1 IV- LI \J 1 1/ 1 1 1 1 L/l VJ VCI 1 IC 1 1 L 


7.08 


7.23 


6.62 


Mprlir^tinn prrnr<; 

i vicu i v_a liui i ci i ui j 


7.28 


7.33 


6.92 


Cntir^l inrirlpirK 

V 1 1 LI V_CI 1 IIICIUCIILj 


7.39 


7.5 


6.92 
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7 13 
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7 OR 

I ,uo 


fi fi7 
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Prpcciirp iilrprc 
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u. :?u 


u.uu 
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Pain 


6.97 


6.58 


7.17 


Patient level measures — satisfaction and perceptions 








Patient and caregiver satisfaction (follow-up during interventions and/or after) 


7.52 


7.69 


7.54 


Activities of daily living/social activities 


7.13 


7.08 


6.67 


Quality of care 


6.92 


6.58 


7 


Provider level measures — satisfaction and perceptions 








Communication between shift staff/optimal choice of communication/information transfer 


7.21 


7.54 


6.23 


Communication and collaboration of physician consultations 


6.94 


5.85 


7.31 



ED, Emergency Department; Ql, quality improvement. 



primary healthcare provider, discharge summary all 
medications and follow-up appointments given to 
patient, readmission rates within 90 days, multiple 
psychiatric readmissions in 30 days); one patient 
outcome measure (mortality/death/cause of death/risk 
of death within 72 h); and one patient satisfaction and 
perceptions measure (patient and caregiver satisfaction 
follow-up during interventions and/or after). These 
measures were all associated with posthospital dis- 
charge and warrant further attention in future 
research to determine what measures should be 
included and/or developed to capture measures that 
hospitals could use to assess the quality of care transi- 
tions with complex-care patients. 



IMPLICATIONS FOR PRACTICE AND POLICY 

To our knowledge, there has been no previous 
research in the area of quality measures for care tran- 
sitions involving complex-care patients across the con- 
tinuum of care. As a next step, further validation of 
these measures is recommended with other key stake- 
holders. This work will require: (1) testing of whether 
these measures actually measure the quality of care (vs 
differences in case mix); (2) determining if there are 
sufficient numbers to enable stable estimates when 
measuring at the provider/hospital level; (3) determin- 
ing the need for risk adjustment and (4) ascertaining 
the level of measurement for the indicators. There is 
potential for these measures to be used as indicators 
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Table 3 Results from postpanel questionnaire of measures achieving at least 75% consensus 




Mean 


Mean 


Mean 


Measure 


total score 


drive Ql 


public reporting 


Prationt 2nn rraranivor craticfrartinn i fr\ r\\/\/_iin Hiirinn inton/ontinnc 2nn/nr raTtari 
rdUcllL dllU Ldlcyivcl bdUbldLUUI 1 V 1 UMUVV U[J UUMIiy II 1 Lcl Vcl 1 LIUI lb dllU/UI dILcly 


O.U J 


o.ur 


0 

o 


RqdH m ice inn r:atac \A/itnin naic 
rvcdUl 1 HbblUI 1 IdLcb VVILIIIII jU Udyb 


7 Rfi 

/ .00 


O. I 0 


7 ^ 


MpHir^tinn rprnnrili^tinn rnmnlptpH 3t ^Hmkcinn 3nH nrinr tn Hicrhpirnp 

IVICUILdLIUI 1 1 CLUI ILI Mu LIUI 1 LUI MjJICLCU dL dUM II jjIUI 1 Cll IU [Jl IUI LU UljUICliyC 


7.82 


8.45 


7.18 


nicrharnp ciimman/ ^all mpHiratinnc anH fnlln\A/-i in annnintmontc^ ni\/pn tn nrimarw hpalthrarp nrn\/iHor 
UlbUldiytr bUlllllldly \a\\ 1 1 IcUILdLIUI 1 b dllU IUIIUVV U|J d|J|JUII IU 1 Icl LjlVtMl LU (Jllllldly 1 Icdlll ILdlc [JIUVIUcI 


7 7 
/ . / 


R 1 


7 3 


RppiHini^^inri ratp*; within 11 h 

i \cau 1 1 ii j jivji i laicj vviLiiiii / t- 1 1 


7.55 


7.82 


7.27 


Rp^Hmkdnn ratpc; within QD Have. 

i \cau 1 1 ii jjiui i laicj vviliiiii j\j uayj 


7.45 


7.91 


7 


Timp frnm Hicrharnp tn hnmprarp niircinn \/kit fnr h inh-rick - natipntc 

lllllc IIUIII UlbUldiyc LU IIUIIICLdlC MUlblliy VlblL IUI lliyil IIjI\ jJdLICllLb 


7.41 


7.91 


Q1 

U..7 1 


Mortality/death/cause of death/risk of death within 72 h 


7.41 


7.55 


7.27 


Discharge Summary (all medications and follow-up appointments) given to patient 


7.35 


7.9 


6.8 


Primary care visit within 7 days postdischarge for high-risk patients 


7.27 


7.64 


6.91 


Multiple psychiatric readmissions in 30 days 


6.95 


7.18 


6.73 



Ql, quality improvement. 



of overall care quality related to care transitions 
involving complex-care patients across the continuum 
of care. 

There is an increasing focus on moving a patient 
out of the hospital system to alleviate wait times and 
in an effort to provide patients with more appropriate 
care in the right settings, such as home-based care and 
community-based programmes and services. However, 
in order to gain the confidence of patients and their 
caregivers in these movements across settings, it is 
essential that organisations be accountable for provid- 
ing and monitoring high-quality transitions. Having 
measures such as these can allow organisations to 
ascertain how well they are doing, make improve- 
ments where necessary and be accountable to the 
patients who they are transitioning. We acknowledge 
that this study has been conducted in the Canadian 
healthcare system, therefore, generalisablity to other 
countries and contexts may not apply exactly. 
However, the Delphi technique described here can be 



replicated in other countries in order to understand 
whether these measures are consistent in other 
contexts. 

Moreover, there is broad applicability in the major- 
ity of the final quality measures selected as they use 
data that is already commonly collected by hospitals 
and other healthcare organisations and could feasibly 
be collected and reported. Some specific points made 
by panel members during the discussion merit 
emphasis. For example, the recommendation to 
further develop measures that are reflective of 
complex-care patient and family members' satisfaction 
and experiences along the care continuum warrants 
further attention. This finding is consistent with the 
view that patients and their families play an integral 
role in ensuring they receive safe care, as they are the 
one constant in care transitions processes. 5 This work 
can draw from existing measures such as the Care 
Transitions Measure used to measure satisfaction with 
discharge from hospital to home. 1 5 56 



Table 4 Final rankings 



Measure 


Ranking 


Mean score 


Readmission rates within 30 days 


1 


3 


Primary care visit within 7 days postdischarge for high-risk* patients 


2 


3.6 


Medication reconciliation completed at admission and prior to discharge 


3 


3.8 


Readmission rates within 72 h 


4 


4.5 


Time from discharge to home care nursing visit for high-risk patients 


5 


4.8 


Discharge summary (all medications and follow-up appointments) given to primary healthcare provider 


6 


5.6 


Discharge summary (all medications and follow-up appointments) given to patient 


7 


6 


Patient and caregiver satisfaction (follow up during interventions and/or after) 


8 


7.5 


Readmission rates within 90 days 


9 


8.6 


Mortality/death/cause of death/risk of death within 72 ht 


10 


9 


Multiple psychiatric readmissions in 30 days 


11 


9.6 



*The panel defined high-risk patients as those who were at high risk for being readmitted to hospital due to the complexity of their condition/disease and 
ability to adhere to and manage their care postdischarge. 

tThe panel defined mortality related to the complexity of their condition/disease with 3 days (72 h) of discharge from hospital. 
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Strengths and limitations 

Engaging key stakeholders in quality measure develop- 
ment is paramount. Important attributes of high- 
quality measures are their relevance to the selected 
problem and field of application, their feasibility and 
their reliability. 10 It is important to note that the 
inclusion of a patient representative on the panel is 
important, as a recent systematic review revealed that 
the participation of patients in quality indicator devel- 
opment is extremely uncommon. 10 Our final panel 
consisted of individuals (researchers, front-line clini- 
cians and a patient) with broad professional represen- 
tation as well as consultants from stakeholder groups 
(national accreditation body and provincial quality 
council). Additionally, our panel included individuals 
from different healthcare systems and sectors increas- 
ing the potential for generalisability of our findings. 

The literature review that was included as a summary 
in the background information and anonymous survey 
used in our process ensured that all members were 
entering the discussion with balanced background and 
content knowledge. We also included open-ended 
questions in the presurvey to provide an opportunity 
for panel members to comment on the list of indicators 
and submit additional indicators. Additionally, an 
experienced health services researcher facilitated the 
discussion and maintained equality among all panel 
members. The anonymous end-of-day survey verified 
that all panelists were comfortable with the process. 

Furthermore, panel discussion was also focused on 
finding important and relevant measures that could act 
as quality indicators in addition to the ones initially 
presented by the research team. This open-ended dis- 
cussion was presented to the panel prior to the 
face-to-face meeting to provide opportunity for pane- 
lists to review and reflect on the measures presented 
and potential missing measures reflective of quality 
care transitions for complex-care patients across the 
care continuum. Upon review of the measures pre- 
meeting and postmeeting, it was evident that the 
face-to-face discussion was a vital component to our 
process as the measures and parameters were redefined 
through the modified Delphi process. This is an 
important finding that should guide future quality per- 
formance measures — research and development. 

There are limitations associated with our modified 
Delphi panel process. First, the review of literature was 
not a systematic review, rather it was a targeted review 
that was designed by researchers with expertise in per- 
formance measures and care transitions. We drew on the 
expertise of other researchers, clinicians and researchers 
that were part of the panel to ensure that key measures 
were not omitted. Second, there was the potential for 
selection bias inherent in panel member selection. To 
minimise selection bias, the research team used a similar 
process used by one of the researchers 17 18 and solicited 
suggestions from unbiased third party individuals, stake- 
holder groups as well as national patient safety 



organisations. However, our initial international panels 
were not able to participate. Thus, some important per- 
spectives may not have been identified. Another 
measure to reduce the risk of selection bias was the use 
of predesigned forms containing rating and selection cri- 
teria during systematic consensus processes. 10 Third, the 
complex nature around the validity of the measures and 
whether they are measuring quality of care and the feasi- 
bility of collecting data on the identified measures is 
another limitation. In our study, panelists were not asked 
to limit their selection to quality measures that could be 
applied immediately with commonly available data. The 
use of administrative data for any quality measures 
always has the limitation of clinical application, as it is 
only a part of a patient's clinical story. Fourth, our panel 
had only one patient representative, and although they 
were a leader of a national patient safety advocacy 
organisation, their views may not be completely repre- 
sentative of patients and caregivers. 

CONCLUSION 

Quality measures are important tools used for 
accountability and QI in healthcare systems. The 
modified Delphi consensus panel approach applied in 
this study enabled us to garner a consensus on mea- 
sures in an important area — care transitions. Although 
a number of these measures are not new and are used 
for the evaluation of other healthcare areas, this study 
provides evidence and support that these measures 
may also be effective for evaluating the quality of 
care transitions. This is the first study focused on care 
transitions and performance measures, thus, more 
research and testing of these measures using a rigorous 
research design are needed to determine their actual 
usefulness and feasibility. Further work is required 
to create standardised operational definitions, for 
example, the patient and caregiver satisfaction 
follow-up during interventions and/or after measure. 

Collectively, future research efforts will produce 
findings that can be used by healthcare administrators, 
clinicians and policy makers to monitor care transi- 
tions and to work together to improve the quality 
of transitions of care with complex-care patients. 
Moreover, individual institutions can evaluate inter- 
ventions aimed at enhancing care transitions for 
complex-care patents' efforts, and provide healthcare 
systems with a consistent means to compare pro- 
gramme effectiveness across institutions. A longer- 
term goal is that this work will inform the 
development of a reporting standard of measures that 
demonstrate quality outcomes associated with transi- 
tions of care with complex-care patients. 
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